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ABSTRACT 

In his pioneer investigations of deductive logical 
reasoning competence, R. H, Ennis (R . H. Ennis and D. H. Paulus, 
1965) used a multiple-choice format in which the premises are given, 
and it is asked whether the conclusion would then be true. In the 
adaptation of his work for use in Jamaica, the three possible answers 
were stated as "yes" (it must be true) , "no" (it can not be true 
based on what you are told) , and "maybe" (it may be true or false) . 
In the original investigations, the results were of no consequence 
for those tested, but in Jamaica questions have been part of 

examinations of importance to the subjects. For this reason, the 
possible effect of question format was investigated by replacing 
"maybe" with "not necessarily." Results with 537 subjects in 1990 
(using "maybe" test answers) and 474 subjects in 1991 (using "not 
necessarily" test answers) indicate that the change of format makes 
no difference to 18 valid items, but results in significant 
differences in response pattern for 13 of 18 invalid items, with more 
correct responses in most cases. Three tables present study data. 
(SLD) 
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The central notion of deductive logic is that of a valid 
argument, an argument in which the conclusion really does 
follow from the premisses. The standard informal explanation 
of validity is that it should be impossible for the premisses 
of the argument to be true and the conclusion false. 

Investigation of people's actual competence in matters of 
deductive logic could use questions of the form "Does r 
follow from p and g?" But given the standard informal 
explanation, one could avoid any uncertainties people might 
feel about what it is for one statement to follow from 
another, by framing the question in terms of the notions of 
truth and falsehood. So, for instance, in his pioneer 
investigations of deductive logical reasoning competence, 
Robert Ennis (Ennis and Paulus, 1965) employed the following 
question structure: 

Suppose you know that p, q , .... 

Then would it be true that r? 
In an adaptation of Ennis' work for use in Jamaica, the three 
possible answers offered (Yes; No; Maybe) are glossed as 
"Yes" means "It must be true, given what you are told"; "No" 
means "It can't be true, given what you are told"; and 
"Maybe" means "It may be true or it may be false. You 
haven't been told enough to be certain whether it is Yes or 
No." 

Given Ennis' question format, and the standard construal 
of validity, when the sentences constitute a deductively 
valid argument the correct answer is either "Yes" or "No"; 
when they do not make up a valid argument the correct answer 
is "Maybe." 

In the course of the Jamaican investigations (reported in 
Nolan and Brandon, 1986, and Brandon, 1990) a doubt arose 
concerning the question format. While the correct answers in 
the case of valid arguments seem conversationally 
appropriate, this does not seem so obviously the case for the 
invalid ones. Contrast these two dialogues: 

(i) Suppose some vegetarians drink milk; would some 
people who drink milk be vegetarians? 
Yes. 
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(ii) Suppose most teachers are women; would it be true 
that most women are teachers? 
Maybe. 

Speaking personally, my usual response in cases such as (ii) 
would be to say "No," or more fully "No, not necessarily." 

In Ennis' original investigation, the results were of no 
consequence for those tested; subjects had no time limit to 
complete the questions; and they were reminded of the meaning 
of the answers on every page of the question booklet. The 
collection of most of the data in Jamaica has involved three 
serious departures from this set-up: the questions have 
formed part of an entrance examination for the Faculty of 
Education; there has been a time-limit for the examination; 
and instructions were given only on the first page of the 
appropriate section of the booklet. With such an increase in 
pressure, it is likely that interference from linguistically 
odd constructions would be more serious than in Ennis' 
original investigations . 

It was decided to investigate the possible effect of the 
question format by using the same items in two successive 
years with one small change in question format: to replace 
"Maybe" with "Not necessarily." 

The item analyses in Tables 1 and 2 report the main 
findings: in general the change of format makes no difference 
to performance on the valid items (Table 1) but a 
considerable difference to performance on the invalid 
questions (Table 2) . The tables give the numbers offering 
each of the three possible answers or omitting to answer the 
question, the correct answer in each case, the difficulty 
index for the item (really a facility index) , and finally the 
chi-square value and its probability for the comparison of 
the distribution for the two years. 

Table 1 shows that in only one of the 18 valid items in 
the test was there a significantly different distribution of 
answers between the two years, though it made no difference 
to the difficulty of the item. In general the three forms of 
reasoning were of similar difficulty for the two groups. 

Comparison with Table 2 shows that the overall facility 
of two of the three invalid types of item increased markedly 
and that in 13 of the 18 invalid items the distribution of 
answers was significantly different. 

These results seem to show that the change from "Maybe" 
to "Not necessarily" has a marked effect in many of those 
cases where it is the salient and correct answer. In most 
cases, more respondents give the correct answer with "Not 
necessarily" as the prompt. They suggest also the 



possibility that the low scores found by Ennis on invalid 
items might to some small extent be a product of the test 
format. 

While performance seems to improve on the invalid items 
taken individually, subjects do not seem to improve quite so 
much when one looks at the consistency of their performance 
on groups of items. Ennis characterises mastery of a 
principle as the correct answering of at least 5 out of the 6 
items for that principle; "borderline" performance involves 
getting 4 of the 6 right. Table 3 shows percentage mastery 
and borderline performance on the six principles tested for 
the two groups. As can be seen, there is some improvement in 
mastery and borderline rates for two of the three principles, 
but there have been smaller fluctuations between years and 
there is a trend for performance on minval to improve over 
the years. To some extent this failure of better performance 
on individual items to translate into mastery is due to the 
time constraints - Tables 1 and 2 reveal increasing omissions 
towards the end of the test. But it is also a matter of what 
is still a hazy grasp of the logic - less than half the 
subjects get right answers on most of the items. 
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Table 1: Item analysis 1990/1 
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argument tested: inodpon is if p then g, p, so g; modtol is if 
p then g, not g, so not p; and dissyl is either p or g, not 
p, so g. The difficulty index above each group of items is 
the mean for that group. 
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Table 2: Item analysis 1990/1 - Invalid items 
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: see the note to table 1; 
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is the principle most A 





are B, all C are A, so some C are B; minval is most A are B, 
most B are C, so most A are C; nonono is no A are B, no B are 
C, so no A are C, 
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Table 3: Percentage mastery/borderline performance 

1990 

MODPON MODTOL DISSYL UMMOST NONONO MINVAL 
72 M 11 8 7 18 
15 21 9 15 6 12 
1991 

MODPON MODTOL DISSYL UMMOST NONONO MINVAL 
74 43 78 6 9 24 
13 20 10 12 10 17 
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